{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "78b45321-7740-4399-b2ad-459811131de3",
      "metadata": {},
      "source": [
        "# How to get log probabilities\n",
        "\n",
        ":::info Prerequisites\n",
        "\n",
        "This guide assumes familiarity with the following concepts:\n",
        "\n",
        "- [Chat models](/docs/concepts/chat_models)\n",
        "\n",
        ":::\n",
        "\n",
        "Certain chat models can be configured to return token-level log probabilities representing the likelihood of a given token. This guide walks through how to get this information in LangChain."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7f5016bf-2a7b-4140-9b80-8c35c7e5c0d5",
      "metadata": {},
      "source": [
        "## OpenAI\n",
        "\n",
        "Install the `@langchain/openai` package and set your API key:\n",
        "\n",
        "```{=mdx}\n",
        "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n",
        "import Npm2Yarn from \"@theme/Npm2Yarn\";\n",
        "\n",
        "<IntegrationInstallTooltip></IntegrationInstallTooltip>\n",
        "\n",
        "<Npm2Yarn>\n",
        "  @langchain/openai @langchain/core\n",
        "</Npm2Yarn>\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f88ffa0d-f4a7-482c-88de-cbec501a79b1",
      "metadata": {},
      "source": [
        "For the OpenAI API to return log probabilities, we need to set the `logprobs` param to `true`. Then, the logprobs are included on each output [`AIMessage`](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html) as part of the `response_metadata`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "d1bf0a9a-e402-4931-ab53-32899f8e0326",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  {\n",
              "    token: \u001b[32m\"Thank\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.70174205\u001b[39m,\n",
              "    bytes: [ \u001b[33m84\u001b[39m, \u001b[33m104\u001b[39m, \u001b[33m97\u001b[39m, \u001b[33m110\u001b[39m, \u001b[33m107\u001b[39m ],\n",
              "    top_logprobs: []\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" you\"\u001b[39m,\n",
              "    logprob: \u001b[33m0\u001b[39m,\n",
              "    bytes: [ \u001b[33m32\u001b[39m, \u001b[33m121\u001b[39m, \u001b[33m111\u001b[39m, \u001b[33m117\u001b[39m ],\n",
              "    top_logprobs: []\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" for\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.000004723352\u001b[39m,\n",
              "    bytes: [ \u001b[33m32\u001b[39m, \u001b[33m102\u001b[39m, \u001b[33m111\u001b[39m, \u001b[33m114\u001b[39m ],\n",
              "    top_logprobs: []\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" asking\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.0000013856493\u001b[39m,\n",
              "    bytes: [\n",
              "       \u001b[33m32\u001b[39m,  \u001b[33m97\u001b[39m, \u001b[33m115\u001b[39m,\n",
              "      \u001b[33m107\u001b[39m, \u001b[33m105\u001b[39m, \u001b[33m110\u001b[39m,\n",
              "      \u001b[33m103\u001b[39m\n",
              "    ],\n",
              "    top_logprobs: []\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\"!\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.00030102333\u001b[39m,\n",
              "    bytes: [ \u001b[33m33\u001b[39m ],\n",
              "    top_logprobs: []\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const model = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  logprobs: true,\n",
        "});\n",
        "\n",
        "const responseMessage = await model.invoke(\"how are you today?\");\n",
        "\n",
        "responseMessage.response_metadata.logprobs.content.slice(0, 5);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d1ee1c29-d27e-4353-8c3c-2ed7e7f95ff5",
      "metadata": {},
      "source": [
        "And are part of streamed Message chunks as well:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "4bfaf309-3b23-43b7-b333-01fc4848992d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[]\n",
            "[\n",
            "  {\n",
            "    token: \"Thank\",\n",
            "    logprob: -0.23375113,\n",
            "    bytes: [ 84, 104, 97, 110, 107 ],\n",
            "    top_logprobs: []\n",
            "  }\n",
            "]\n",
            "[\n",
            "  {\n",
            "    token: \"Thank\",\n",
            "    logprob: -0.23375113,\n",
            "    bytes: [ 84, 104, 97, 110, 107 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" you\",\n",
            "    logprob: 0,\n",
            "    bytes: [ 32, 121, 111, 117 ],\n",
            "    top_logprobs: []\n",
            "  }\n",
            "]\n",
            "[\n",
            "  {\n",
            "    token: \"Thank\",\n",
            "    logprob: -0.23375113,\n",
            "    bytes: [ 84, 104, 97, 110, 107 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" you\",\n",
            "    logprob: 0,\n",
            "    bytes: [ 32, 121, 111, 117 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" for\",\n",
            "    logprob: -0.000004723352,\n",
            "    bytes: [ 32, 102, 111, 114 ],\n",
            "    top_logprobs: []\n",
            "  }\n",
            "]\n",
            "[\n",
            "  {\n",
            "    token: \"Thank\",\n",
            "    logprob: -0.23375113,\n",
            "    bytes: [ 84, 104, 97, 110, 107 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" you\",\n",
            "    logprob: 0,\n",
            "    bytes: [ 32, 121, 111, 117 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" for\",\n",
            "    logprob: -0.000004723352,\n",
            "    bytes: [ 32, 102, 111, 114 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" asking\",\n",
            "    logprob: -0.0000029352968,\n",
            "    bytes: [\n",
            "       32,  97, 115,\n",
            "      107, 105, 110,\n",
            "      103\n",
            "    ],\n",
            "    top_logprobs: []\n",
            "  }\n",
            "]\n",
            "[\n",
            "  {\n",
            "    token: \"Thank\",\n",
            "    logprob: -0.23375113,\n",
            "    bytes: [ 84, 104, 97, 110, 107 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" you\",\n",
            "    logprob: 0,\n",
            "    bytes: [ 32, 121, 111, 117 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" for\",\n",
            "    logprob: -0.000004723352,\n",
            "    bytes: [ 32, 102, 111, 114 ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \" asking\",\n",
            "    logprob: -0.0000029352968,\n",
            "    bytes: [\n",
            "       32,  97, 115,\n",
            "      107, 105, 110,\n",
            "      103\n",
            "    ],\n",
            "    top_logprobs: []\n",
            "  },\n",
            "  {\n",
            "    token: \"!\",\n",
            "    logprob: -0.00039694557,\n",
            "    bytes: [ 33 ],\n",
            "    top_logprobs: []\n",
            "  }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "let count = 0;\n",
        "const stream = await model.stream(\"How are you today?\");\n",
        "let aggregateResponse;\n",
        "\n",
        "for await (const chunk of stream) {\n",
        "  if (count > 5) {\n",
        "    break;\n",
        "  }\n",
        "  if (aggregateResponse === undefined) {\n",
        "    aggregateResponse = chunk;\n",
        "  } else {\n",
        "    aggregateResponse = aggregateResponse.concat(chunk);\n",
        "  }\n",
        "  console.log(aggregateResponse.response_metadata.logprobs?.content);\n",
        "  count++;\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3c5222d2",
      "metadata": {},
      "source": [
        "## `topLogprobs`\n",
        "\n",
        "To see alternate potential generations at each step, you can use the `topLogprobs` parameter:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "fa4d38b1",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  {\n",
              "    token: \u001b[32m\"I'm\"\u001b[39m,\n",
              "    logprob: \u001b[33m-2.2864406\u001b[39m,\n",
              "    bytes: [ \u001b[33m73\u001b[39m, \u001b[33m39\u001b[39m, \u001b[33m109\u001b[39m ],\n",
              "    top_logprobs: [\n",
              "      {\n",
              "        token: \u001b[32m\"Thank\"\u001b[39m,\n",
              "        logprob: \u001b[33m-0.28644064\u001b[39m,\n",
              "        bytes: [ \u001b[33m84\u001b[39m, \u001b[33m104\u001b[39m, \u001b[33m97\u001b[39m, \u001b[33m110\u001b[39m, \u001b[33m107\u001b[39m ]\n",
              "      },\n",
              "      {\n",
              "        token: \u001b[32m\"Hello\"\u001b[39m,\n",
              "        logprob: \u001b[33m-2.0364406\u001b[39m,\n",
              "        bytes: [ \u001b[33m72\u001b[39m, \u001b[33m101\u001b[39m, \u001b[33m108\u001b[39m, \u001b[33m108\u001b[39m, \u001b[33m111\u001b[39m ]\n",
              "      },\n",
              "      { token: \u001b[32m\"I'm\"\u001b[39m, logprob: \u001b[33m-2.2864406\u001b[39m, bytes: [ \u001b[33m73\u001b[39m, \u001b[33m39\u001b[39m, \u001b[33m109\u001b[39m ] }\n",
              "    ]\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" just\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.14442946\u001b[39m,\n",
              "    bytes: [ \u001b[33m32\u001b[39m, \u001b[33m106\u001b[39m, \u001b[33m117\u001b[39m, \u001b[33m115\u001b[39m, \u001b[33m116\u001b[39m ],\n",
              "    top_logprobs: [\n",
              "      {\n",
              "        token: \u001b[32m\" just\"\u001b[39m,\n",
              "        logprob: \u001b[33m-0.14442946\u001b[39m,\n",
              "        bytes: [ \u001b[33m32\u001b[39m, \u001b[33m106\u001b[39m, \u001b[33m117\u001b[39m, \u001b[33m115\u001b[39m, \u001b[33m116\u001b[39m ]\n",
              "      },\n",
              "      { token: \u001b[32m\" an\"\u001b[39m, logprob: \u001b[33m-2.2694294\u001b[39m, bytes: [ \u001b[33m32\u001b[39m, \u001b[33m97\u001b[39m, \u001b[33m110\u001b[39m ] },\n",
              "      {\n",
              "        token: \u001b[32m\" here\"\u001b[39m,\n",
              "        logprob: \u001b[33m-4.0194297\u001b[39m,\n",
              "        bytes: [ \u001b[33m32\u001b[39m, \u001b[33m104\u001b[39m, \u001b[33m101\u001b[39m, \u001b[33m114\u001b[39m, \u001b[33m101\u001b[39m ]\n",
              "      }\n",
              "    ]\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" a\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.00066632946\u001b[39m,\n",
              "    bytes: [ \u001b[33m32\u001b[39m, \u001b[33m97\u001b[39m ],\n",
              "    top_logprobs: [\n",
              "      { token: \u001b[32m\" a\"\u001b[39m, logprob: \u001b[33m-0.00066632946\u001b[39m, bytes: [ \u001b[33m32\u001b[39m, \u001b[33m97\u001b[39m ] },\n",
              "      {\n",
              "        token: \u001b[32m\" lines\"\u001b[39m,\n",
              "        logprob: \u001b[33m-7.750666\u001b[39m,\n",
              "        bytes: [ \u001b[33m32\u001b[39m, \u001b[33m108\u001b[39m, \u001b[33m105\u001b[39m, \u001b[33m110\u001b[39m, \u001b[33m101\u001b[39m, \u001b[33m115\u001b[39m ]\n",
              "      },\n",
              "      { token: \u001b[32m\" an\"\u001b[39m, logprob: \u001b[33m-9.250667\u001b[39m, bytes: [ \u001b[33m32\u001b[39m, \u001b[33m97\u001b[39m, \u001b[33m110\u001b[39m ] }\n",
              "    ]\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" computer\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.015423919\u001b[39m,\n",
              "    bytes: [\n",
              "       \u001b[33m32\u001b[39m,  \u001b[33m99\u001b[39m, \u001b[33m111\u001b[39m, \u001b[33m109\u001b[39m,\n",
              "      \u001b[33m112\u001b[39m, \u001b[33m117\u001b[39m, \u001b[33m116\u001b[39m, \u001b[33m101\u001b[39m,\n",
              "      \u001b[33m114\u001b[39m\n",
              "    ],\n",
              "    top_logprobs: [\n",
              "      {\n",
              "        token: \u001b[32m\" computer\"\u001b[39m,\n",
              "        logprob: \u001b[33m-0.015423919\u001b[39m,\n",
              "        bytes: [\n",
              "           \u001b[33m32\u001b[39m,  \u001b[33m99\u001b[39m, \u001b[33m111\u001b[39m, \u001b[33m109\u001b[39m,\n",
              "          \u001b[33m112\u001b[39m, \u001b[33m117\u001b[39m, \u001b[33m116\u001b[39m, \u001b[33m101\u001b[39m,\n",
              "          \u001b[33m114\u001b[39m\n",
              "        ]\n",
              "      },\n",
              "      {\n",
              "        token: \u001b[32m\" program\"\u001b[39m,\n",
              "        logprob: \u001b[33m-5.265424\u001b[39m,\n",
              "        bytes: [\n",
              "           \u001b[33m32\u001b[39m, \u001b[33m112\u001b[39m, \u001b[33m114\u001b[39m, \u001b[33m111\u001b[39m,\n",
              "          \u001b[33m103\u001b[39m, \u001b[33m114\u001b[39m,  \u001b[33m97\u001b[39m, \u001b[33m109\u001b[39m\n",
              "        ]\n",
              "      },\n",
              "      {\n",
              "        token: \u001b[32m\" machine\"\u001b[39m,\n",
              "        logprob: \u001b[33m-5.390424\u001b[39m,\n",
              "        bytes: [\n",
              "           \u001b[33m32\u001b[39m, \u001b[33m109\u001b[39m,  \u001b[33m97\u001b[39m,  \u001b[33m99\u001b[39m,\n",
              "          \u001b[33m104\u001b[39m, \u001b[33m105\u001b[39m, \u001b[33m110\u001b[39m, \u001b[33m101\u001b[39m\n",
              "        ]\n",
              "      }\n",
              "    ]\n",
              "  },\n",
              "  {\n",
              "    token: \u001b[32m\" program\"\u001b[39m,\n",
              "    logprob: \u001b[33m-0.0010724656\u001b[39m,\n",
              "    bytes: [\n",
              "       \u001b[33m32\u001b[39m, \u001b[33m112\u001b[39m, \u001b[33m114\u001b[39m, \u001b[33m111\u001b[39m,\n",
              "      \u001b[33m103\u001b[39m, \u001b[33m114\u001b[39m,  \u001b[33m97\u001b[39m, \u001b[33m109\u001b[39m\n",
              "    ],\n",
              "    top_logprobs: [\n",
              "      {\n",
              "        token: \u001b[32m\" program\"\u001b[39m,\n",
              "        logprob: \u001b[33m-0.0010724656\u001b[39m,\n",
              "        bytes: [\n",
              "           \u001b[33m32\u001b[39m, \u001b[33m112\u001b[39m, \u001b[33m114\u001b[39m, \u001b[33m111\u001b[39m,\n",
              "          \u001b[33m103\u001b[39m, \u001b[33m114\u001b[39m,  \u001b[33m97\u001b[39m, \u001b[33m109\u001b[39m\n",
              "        ]\n",
              "      },\n",
              "      {\n",
              "        token: \u001b[32m\"-based\"\u001b[39m,\n",
              "        logprob: \u001b[33m-6.8760724\u001b[39m,\n",
              "        bytes: [ \u001b[33m45\u001b[39m, \u001b[33m98\u001b[39m, \u001b[33m97\u001b[39m, \u001b[33m115\u001b[39m, \u001b[33m101\u001b[39m, \u001b[33m100\u001b[39m ]\n",
              "      },\n",
              "      {\n",
              "        token: \u001b[32m\" algorithm\"\u001b[39m,\n",
              "        logprob: \u001b[33m-10.626073\u001b[39m,\n",
              "        bytes: [\n",
              "           \u001b[33m32\u001b[39m,  \u001b[33m97\u001b[39m, \u001b[33m108\u001b[39m, \u001b[33m103\u001b[39m,\n",
              "          \u001b[33m111\u001b[39m, \u001b[33m114\u001b[39m, \u001b[33m105\u001b[39m, \u001b[33m116\u001b[39m,\n",
              "          \u001b[33m104\u001b[39m, \u001b[33m109\u001b[39m\n",
              "        ]\n",
              "      }\n",
              "    ]\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const modelWithTopLogprobs = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  logprobs: true,\n",
        "  topLogprobs: 3,\n",
        "});\n",
        "\n",
        "const res = await modelWithTopLogprobs.invoke(\"how are you today?\");\n",
        "\n",
        "res.response_metadata.logprobs.content.slice(0, 5);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "19766435",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "You've now learned how to get logprobs from OpenAI models in LangChain.\n",
        "\n",
        "Next, check out the other how-to guides chat models in this section, like [how to get a model to return structured output](/docs/how_to/structured_output) or [how to track token usage](/docs/how_to/chat_token_usage_tracking)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Deno",
      "language": "typescript",
      "name": "deno"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",
      "pygments_lexer": "typescript",
      "version": "5.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
